Using the Picture Object

The Picture object in Advanced Process Automation enables you to capture text from images using the Optical Recognition Method (OCR). The Picture object uses the Nicomsoft OCR engine.

You can capture text from a currently open page in a PDF document, images (JPG, BMP, TIFF, GIF), or the user interface of a remote application that is running (for example, a Citrix session).

OCR functionality is included in the Picture object, which can be found in the Direct.Vsd.Library. See Picture Object Functionality.

OCR supports the following languages:

Bulgarian

Catalan

Croatian

Czech

Danish

Dutch

English

Estonian

Finnish

French

German

Hungarian

Indonesian

Italian

Latvian

Lithuanian

Norwegian

Polish

Portuguese

Romanian

Russian

Slovak

Slovenian

Spanish

Swedish

Turkish

Picture Object Functionality

You can review the available functions of the Picture object from the Direct.Vsd.Library in Real-Time Designer.

To view Direct.Vsd.Library functionality:

1.

In Real-Time Designer, open the Project tab .

2.

Under the References section, expand the Library References node, and select Direct.Vsd.Library.

3.

Select the Functionality tab.

4.

From the Type drop-down list, select Picture.

The following properties are available:

Property	Description
Block Count	Gets the number of text blocks (OCR) from this picture.
Height	Gets the height of this picture (in pixels).
Width	Gets the width of this picture (in pixels).

The following functions are available:

Function	Description
Compare	Compares two pictures.
Copy to Clipboard	Clears the clipboard and then adds a picture in the Bitmap (BMP) format.
Create from Clipboard	Creates a picture from the clipboard.
Create from File	Creates a picture from the specified image file.
Create from Window	Brings the specified window into the foreground, captures screenshot from this window and creates a picture.
Crop Image	Crops the image from the source Picture object and retrieves a new Picture object.
Determine Brightness	Determines a brightness value in percentages of a given area of Picture object by coordinates (value 100 is white).
Determine Control Brightness	Determines a brightness value in percentages of a given Screen Element or VSD Control (value 100 is white).
Get Control Image	Captures image from VSD Control or Screen Element and retrieves new Picture object.
Get Difference From Template	Returns a value indicating the difference in brightness between two Picture images. A value of 100 indicates no difference. Anything lower than 100 indicates a difference, the lower the value the greater the difference in brightness. This function is used to determine whether pictures are the sam. Typical use case is to determine whether a checkbox or radio button is selected by comparing a pre-prepared image of the selected checkbox/radio button to the actual one.
Get Image Text	Returns recognized text in the Picture image.
Get OCR Text Block	Returns recognized text from a specified OCR Text Block by its index.
Get Words	Recognizes text and returns VSD Word Data Object with words and rectangles collection.
Recognize Text from Control	Brings the control or parent window into the control foreground, captures a screenshot, crops the image for specified controlm creates a picture, and recognizes text. The control is VSD Control or Screen Element exclude browsers (IE, Chrome, Firefox) controls.
Save	Saves a Picture to the image file according to a specified full path.

The following Scene Composer functions are available:

Control	Function	Description
PictureBox	Get Picture	Returns a Picture object from a PictureBox control of Windows scene.
DataCapture	Capture Images	Returns a Picture objects collection from a DataCapture control of General scene.

Getting Text from an Image

This procedure describes how to get text from an image file (JPG, BMP, TIFF, GIF, PNG) using the Create from File and Get Image Text functions. In this example, you do not have to open the image source file. OCR automatically converts the content of the image file to the Picture object.

The following PNG image is used as the example in the following procedure.

To capture text from an image file:

1.

In the Business Entity tab, create a new Business Entity type. Then create an instance of the New Type.

2.

Add a New Property to the New Type and give it a name (Picture object in the example).

3.

In the Properties tab, open the Property Type drop-down list, and select Library Objects > General > Picture.

4.

Under the New Type, add two instances of New Property of the Text type, and give them names (Text and Source File Path in the example).

5.

Under the New Type, add a New Function, and give it a name (Get Text From File in the example).

6.

In the Function Body section, select Insert Instruction > Select Function > Picture Object > Create from File.

7.

Click <Select File Name…> and select the Source File Path business entity property.

8.

In the Function Body section, select Insert Instruction > Assignment. Click the <Select Object…> link and select the Text property.

9.

Click the <Enter Value…> link and select Picture Object > Get Image Text.

10.

To test the above procedure, open the Presentation tab, and add a New Blank Callout.

11.

In the New Callout, add the link to the Get Text From File function, and define the InputText Box with the assignment to the Source File Path business entity property. Then insert dynamic text with the assignment to the Text business entity property.

12.

In the Project tab, create a new Quick callout.

13.

Save and run the project.

14.

Open the Quick callout, and define the full path with the source image file.

15.

Click the Get Text From File link.

The text from the image is displayed in the callout.

Getting Text from an Application Running in Citrix

You can capture text from an image in a remote Citrix session.

To get text from the specified window (Win32 Form), you can use two methods by using different OCR functions:

Initialize the Picture object using the Create from Window function, and then get the text using the Get Image Text function.

This method is explained the procedure below.

Create a Picture and get the text with the Recognize Text from Control function.

You can also get text from OCR text blocks. In this case you must create a Picture and use the Get OCR Text Block function. To get the amount of OCR blocks, use the Block Count property of the Picture.

To capture text from a remote application running in a Citrix session:

1.

Open a Citrix session and run a test application.

2.

Capture the ICA client window. A Win32 Form screen element is captured.

3.

In the Business Entity tab, create a new Business Entity type.

4.

Create an instance of New Type.

5.

Add a New Property to the New Type and give it a name (Picture object in the example).

6.

In the Properties tab, open the Property Type drop-down list, and select Library Objects > General > Picture.

7.

Under the New Type, add a New Function, and give it a name (Get Text in the example).

8.

In the Function Body section, select Insert Instruction > Select Function > Picture Object > Create from Window.

9.

Click the <Select Window Handle…> link and select a window handle of the captured Win32 Form screen element (Screen Elements > wfica32 > TRANSPARENT WINDOWS CLIENT - AnySoft Wrapper Controls Test Application v1.0 > Window Handle).

10.

Under the New Type, add a new property of the Text type and give it a name (Text in the example).

11.

Select the Get Text function under the New Type, and in the Function Body section, select Insert Instruction > Assignment. Then, click the <Select Object…> link and select Text property.

12.

Click <Enter Value…> and select Picture Object > Get Image Text.

13.

To test the previous steps, click the Presentation tab and add a New Blank Callout.

14.

In the New Callout, add the link to the Get Text function, and insert dynamic text with the assignment to the Text business entity property.

15.

In the Project tab, apply the New Callout to a Quick callout.

16.

Save and run the project. Make sure that the test application is running in the Citrix session.

17.

Open the Quick callout.

Click the Get Text link. The text from the UI of the test application is displayed in the callout.

Getting the Number of Words and Coordinates

The following procedure describes how to count the number of words in the image, and get the coordinates for each word in the image. For example, the following callout shows a list of words captured from an image, as well as the coordinates of each image.

This procedure uses the Get Words function of Picture and the functionality of VSD Word Data Object (Words property and Get Word Rectangle function).

To get the number of words and coordinates from a picture:

1.

This example uses a PDF for the image. Capture the PDF reader window as a screen element using the Win32 connector.

2.

In the Business Entities tab, click New Type to create a new business entity. In the Properties tab, from the Base type drop-down list, select Library Object > General > Picture.

3.

Click New Function to create a new function under the Picture object and give it a name (Get words with coordinate in the example).

4.

Initialize the Picture object.

Click Insert Instruction > Function Invocation > Select Function > Business Entity > Picture Object > Create from Window, and define the Window Handler of the captured window.

5.

To create a list of words, click Insert Instruction > Assignment.

6.

Click New Property to add a new property under the Picture object business entity type. Give it a name (VSD Word Data Object instance in the example). From the Base type drop-down list, select Library Objects > Scene Composer > VSD Word Data Object.

7.

In the Function Body, click Select Object link and add the VSD Word Data Objectinstance business entity property.

8.

Click Enter Value and assign the Get Words function (Business Entity > Picture object).

9.

To get the list of words, click Insert Instruction > Assignment.

10.

Click New Property to add a new property under the Picture object business entity type. Give it a name (Words list in the example). From the Property Type drop-down list, select List of Text.

11.

Select the Picture object and in the Function Body, click Select Object, and the Words list business entity property.

12.

Click Add Text and select the Words property of the VSD Word Data Object (Business Entity > Picture Object > VSD Word Data Object > Words).

13.

To get the number of words, click New Property to add a new property under the Picture object business entity type. Give it a name (Words amount in the example). From the Property drop-down list, select Number.

14.

Highlight the Get words with coordinates function, and in the Function Body, click Insert Instruction > Assignment.

15.

Click Select Object and the Words amount business entity property. Click Enter Value and select the Number of Elements property of the Words list business entity (Business Entity > Picture Object > Words list > Number of elements).

16.

Create a counter to bring the words into a list by declaring a new local variable of Number type. Click Insert Instruction > Local Variable. Then click Select Type > Number. Give the variable a name such as countItem.

17.

To assign the initial value into the counter, click Insert Instruction > Assignment, and assign the value 1 to countItem.

18.

Create a For each loop to get coordinates of each word in the Words list collection. Give a name to the collection's item. Click Insert Instruction > For each. Click Select Collection > Words list, and then click ItemName and name the item.

19.

Click Insert Instruction > and Function Invocation, which will be used to create a list of screen element rectangles. Then click Select Instruction > Business Entities.

20.

Click New Property to add a new property under the Picture object business entity type (List of coordinates). From the Base type drop-down list, select Library Objects > List of Screen Element Rectangle (use the path: Library Objects > General).

21.

In the Functions Body of the Picture object, click Select Function and add the action of the Business Entities > Picture Object > List of coordinates business entity property. Click Add item.

22.

Click Select Business Entity, and assign the Get Word Rectangle function of Business Entities > Picture Object > VSD Word Data Objectinstance business entity.

23.

Click Select Item, and assign countItem.

24.

To create an increment of the countItem, select Insert Instruction > Assignment, and assign the Sum Integers function where the argument value is 1 countItem (Linbrary Objects > Math > Sum Integers).

25.

Create a callout to call the function and to display the text. In the Presentation tab, click New Blank Callout.

26.

In the callout, add a link to the Get words with coordinates business entity function.

Click the Insert a function button, and then Insert Instruction > Function Invocation. Click Select Function > Get words with coordinates. Enter an Action name.

27.

In the callout, create a link to the Get words with coordinates business entity function and give a name to the link.

In the callout, add an InputText Box with the assignment to Words amount business entity property.

Open the Elements drop-down list from the Controls sub tab and select InputText Box. Right-click the fields and select Assignment. In the Text property, select Words amount of Picture object.

28.

To bind a table with the Words list business entity, click the Table drop-down list from the Table sub-tab, and select Insert Table.

29.

Right-click the table and select Assignment. In the Data property, select Words list of Picture object. Format the table and add a title (Words).

30.

To bind a table with the List of coordinates business entity, click the Table drop-down list from the Table sub-tab, and select Insert Table.

31.

Open the Elements drop-down list from the Controls sub tab and select Insert Table. Right-click the field and select Assignment. In the Data property, select Business Entities > List of coordinates of Picture object. Format the table and add a title (Words rectangles) and titles for coordinates.

32.

In the Project tab, create a new Quick callout.

33.

Save and test the project in runtime.

a.

Click Test Project > Run Main Project.

b.

Open the Quick callout and click the function's link. Move the callout to the window from which you want to capture the text.